Computers Si Education 78 (2014) 321-332 



ELSEVIER 


Contents lists available at ScienceDirect 

Computers & Education 

journal homepage: www.elsevier.com/locate/compedu 


Computers 

Education 


The acceptability and efficacy of an intelligent social tutoring system 

Rebecca Polley Sanchez , Chelsea M. Bartel, Emily Brown, Melissa DeRosier 

3C Institute, 1907 North Harrison Ave, Suite 200, Cary, NC 27513, USA 



CrossMark 


ARTICLE INFO 


ABSTRACT 


Article history: 

Received 19 February 2014 
Received in revised form 
18 June 2014 
Accepted 23 June 2014 
Available online 1 July 2014 


Keywords: 

Intelligent tutoring system 
Educational computer game 
eLearning 

Social skills training 
Social skills intervention 


This study tested the acceptability and efficacy of an innovative intelligent tutoring system (ITS), 
Adventures Aboard the S.S. GRIN, that translates the evidence-based in-person Social Skills Group Inter- 
vention ( SSGR1N ) into an interactive game-based social tutorial. This randomized controlled pilot trial 
tested the first half of the social tutorial software for children with social skills challenges. Participating 
children in grades 3-5 were randomly assigned to immediate treatment (n = 19) or wait-list control 
(n = 17). User ratings indicated the software was easy to use and well-liked for this audience. The 
program was also associated with observable changes in social skills and behavior; children who 
interacted with Adventures Aboard the SS. GRIN exhibited lower psychosocial distress and higher 
behavioral and emotional strength at post-test compared to children who did not. This pilot study offers 
important insights about the feasibility and potential effectiveness of online social skills training and lays 
the groundwork for future full-scale effectiveness testing. The advantages of using this state-of-the-art 
approach and its implications for improving social, emotional, and academic outcomes are discussed. 

© 2014 Elsevier Ltd. All rights reserved. 


1. Introduction 

7.1. Social skills are crucial for success 

There is growing recognition that social skills are critical for success both in school and in life. Strong support for social skills education is 
shared by the public, educators, and policymakers alike (Camilli, Vargas, Ryan, & Barnett, 2010; Meier, DiPerna, & Oster, 2006; National 
Association for the Education of Young Children, 2009; National Research Council, 2012). Gallup Poll results indicate that a majority of 
people think these skills should be taught to children (Bushaw & Lopez, 2013), and this opinion is increasingly being reflected in state and 
federal policies (Dusenbury, Zadrizil, Mart, & Weissberg, 2011; HR 1875, 2013). Parents, educators, mental health professionals, and others 
invested in the development of children and youth are seeking tools to promote positive social development. 

There is good reason for this increasing emphasis on social competence. Children who struggle with social skills are at heightened risk for 
a wide array of externalizing and internalizing problems and serious negative outcomes, including aggression, anxiety, bullying, depression, 
and substance abuse, as well as academic failure and school dropout (DeRosier, Kupersmidt, & Patterson, 1994; Dodge & Crick, 1990; French 
& Conrad, 2001; Hawkins, Catalano, & Miller, 1992; Kupersmidt & DeRosier, 2004; Najaka, Gottfredson, & Wilson, 2001; Olweus, 1993; 
Parker, Rubin, Erath, Wojslawowicz, & Buskirk, 2006; Solberg, Olweus, & Endresen, 2007; Spooner, 1999; Weiss & Hechtman, 1993). Un- 
addressed problems with peers tend to worsen over time, and as these problems become more chronic and severe the likelihood of serious 
negative consequences in adolescence significantly increases (DeRosier et al., 1994; Kupersmidt & DeRosier, 2004; Parker et al„ 2006). 


7.2. Social skills training 

Fortunately, social skills training (SST) can improve outcomes for children who struggle. Scores of studies support the efficacy of these 
interventions for improving children's social skills and academic functioning, and for reducing aggression, mental health problems, and 
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substance use (e.g., Beidel, Turner, Young, & Paulson, 2005; DeRosier, 2004; Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011 ; Foster 
& Bussman, 2007; Hawker & Boulton, 2000; Joseph & Strain, 2003; Losel & Beelmann, 2003; Zins, Weissberg, Wang, & Walberg, 2004). 
Effective universal or targeted interventions can help children learn and practice the social skills that increase positive and/or inhibit 
negative social interactions (Asher & Renshaw, 1981 ; Coie, Dodge, & Kupersmidt, 1990; Dodge & Feldman, 1990; Merrell & Gimpel, 1997). For 
example, SSTs directly teach cooperative skills that promote companionship and decrease domineering or impulsive behaviors. Similarly, 
SSTs enhance empathetic behaviors that promote communication and the ability to take another's perspective. 

Unfortunately, the reach of traditional in-person SST is limited (IVlueser & Bellack, 2007). Logistical barriers, such as the need for trained 
professionals to administer the program, financial costs to families, and time and travel requirements for children to participate, mean that 
many children who could benefit from SST never receive intervention. The families of children in need of social intervention may lack the 
means to access such treatment, or be unable to commit the time or resources required for in-person intervention. Even when individuals 
can afford and access traditional SST, in-person treatment delivery often does not provide the repeated opportunities for skill practice and 
concept exposure shown to result in greater intervention benefits (Foster & Bussman, 2007; Greenberg, Domitrovich, & Bumbarger, 2001), 
including extending learning and practice opportunities beyond the group setting (Greenberg et al., 2001). The opportunity to not only 
address these barriers but also offer an unprecedented level of individualized, adaptive intervention is now available, as advances in 
emergent technologies are making it possible to translate evidence-based SST programs into broadly accessible engaging game-like 
platforms. 

1.3. Intelligent tutoring systems for social skills training 

An Intelligent Tutoring System (ITS) is a computer-based learning system that attempts to adapt to the needs of learners (Self, 1999). ITSs 
foster and assess learning through adaptive interaction between the student and the system, and the instruction contains both domain- 
specific pedagogical knowledge and knowledge of the learner (Bordeau & Grandbastien, 2010; Collins & O'Brien, 2011). As the learner 
interacts with the software, the software responds by selecting the most appropriate content and pedagogical style for that learner; in this 
way, the software reacts in much the same way a human tutor would. The active engagement through interactivity between a child and the 
software can enhance learning and memory beyond more passive instructional methods (e.g., Corbett, Koedinger, & Hadley, 2001 ; Graesser, 
Jeon, & Duffy, 2008). 

ITSs offer an innovative way to advance SST, address barriers to effective implementation, and reach children on a much broader scale 
than traditional methodologies. In effect, ITSs for SST have the potential to lower cost, eliminate the need for travel, accommodate any 
schedule, and eliminate inconsistencies in treatment delivery across different providers. Beyond these practical benefits, ITSs can improve 
upon traditional SST models by providing repeated and unlimited practice opportunities within an intelligent, interactive learning envi- 
ronment that can individualize instruction at a level not typically possible with in-person SST. Specifically, ITS software has the capacity to 
adjust to a child's individual needs by varying difficulty based on performance and providing immediate personalized feedback and 
pedagogical assistance to guide the child towards target learning goals (Hutcherson, Langone, Ayers, & Clees, 2004; Mazzotti, Test, Wood, & 
Richter, 2009). 

An additional advantage of interactive game-like ITSs is that they are inherently engaging and motivating for today's children, for whom 
technology has become a ubiquitous part of life (Fitzgerald, 2005; NCREL & Metiri Group, 2003). Learning through virtual or simulated 
environments can be a great motivator in the educational process and a valuable part of learning and development (Joyce, Gerhard, & Debry, 
2009; Ke, 2009). An interactive digital environment lets a child think, understand, prepare, execute actions, and adjust strategy as needed 
(Gee, 2003) by employing simulated situations analogous to those they experience in real life. Employing ITSs for SST offers the opportunity 
to improve technical capability and clinical practice in SST. It also provides a safe practice environment; practicing fledgling social skills with 
peers can be risky, but practicing in the virtual environment gives children a safe place to fail and then try again. 

Although the use of ITSs for academic skills such as math and science is becoming more commonplace, an ITS for SST is highly innovative, 
and as a result the research base is just being established. Early results indicate that interactive software may be an effective tool for 
remediating social skills for children. To date, most of the available research (e.g., Golan & Baron-Cohen, 2006; Sansoti & Powell-Smith, 
2008) has focused on children with autism spectrum disorders, who experience marked social skill deficits, and most of the existing 
programs are simplistic in design and implementation with little to no dynamic adjustment in difficulty or personalized feedback. One 
recent system designed to promote social skills and prevent bullying called Quest for the Golden Rule (Rubin-Vaughan, Pepler, Brown, & Craig, 
2011 ) includes more complex social problem solving situations, and preliminary results look promising. However, because that research did 
not include a control group, it's not possible to draw conclusions about the effectiveness of Quest for the Golden Rule. 

1.4. Adventures aboard the S.S. GRIN 

1.4.1. Social skills group intervention: the evidence-based in-person program 

The primary goal of the interactive program is to enable broad delivery of an evidence-based SST and provide children with a safe and 
private learning environment in which to build and practice social skills through interactive social problem solving (SPS) scenes. The specific 
skills included were based on Social Skills Group Intervention {SSGRIN) f an evidence-based in-person SST program. Developed at 3C Institute 
and currently used with thousands of children across the United States and abroad, SSCRJN has been repeatedly tested resulting in a strong 
empirical base for its effectiveness (e.g., DeRosier, 2004; DeRosier & Gilliom, 2007; DeRosier & Marcus, 2005). 

SSGRIN is a 10 week in-person small group intervention for children with social skills deficits, led by a trained professional (e.g., psy- 
chologist or school counselor). SSGRIN is designed to be broadly applicable to children experiencing a wide range of social difficulties, 
including immature social skills relative to same-aged peers; social isolation with few close friends; rejection, teasing, or bullying by peers; 
social anxiety or awkwardness with peers; aggressive or bullying behavior with peers; and impulsive, disruptive behaviors. The program 
addresses the five social and emotional learning (SEL) competency clusters identified by CASEL (2013), which include self-awareness, self- 
management, social awareness, relationship skills, and responsible decision making. It focuses on increasing children's capacity to evaluate 
social situations and respond in an adaptive, prosocial manner to achieve social goals. Children who participate in SSGRIN show increased 
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social skills knowledge, social problem solving, social acceptance, self-esteem, and emotional functioning, as well as lower depression, 
anxiety, and aggressive behavior (DeRosier, 2004, 2007; DeRosier & Marcus, 2005). 

1.4.2. Program description and development 

To develop the ITS Adventures Aboard the S.S. GRIN, we translated the SSGR1N curriculum into a game-like instructional program by 
designing in-game tasks that required children to apply specific social skills in order to solve each social problem encountered. The 
translation was done by a Developmental Psychologist (Ph.D.) with expertise in children's emotional development, and reviewed and 
approved by the developer of the original curriculum, Dr. Melissa DeRosier. Developed for children in grades 3-5 (aged 8-12 years) who 
struggle with social skills, the game-like program was designed to actively engage children in social problem solving tasks through 
interactive software technology, and reflects best practices for children's educational interactive software (e.g., Garris, Ahlers, & Driskell, 
2002; Mathan & Koedinger, 2002; Roussou, 2005). 

To develop the interactive game Adventures Aboard the S.S. GRIN, we designed the SPS scenes upon the foundations of general problem- 
solving models and cognitive-behavioral therapy (e.g., Bloomquist, 2005), as well as decades of theory and research in children’s social 
development (e.g., Bandura, 1977; Crick & Dodge, 1994; Parker et al., 2006). We incorporated research-supported SST strategies such as 
increasing social knowledge, practice with personalized feedback, and positive reinforcement (Elliott & Gresham, 1993) and incorporating 
behavioral (i.e., what to do or not do in a social situation), cognitive (i.e., assumptions, evaluating consequences, goal setting), and emotional 
(i.e., self-awareness, emotion regulation) social skills into instruction (Asher, Parker, & Walker, 1996; Graczyk et al., 2000). This integrative 
approach helps children think through a given social situation, consider alternative social problem solving approaches, and evaluate the 
success of a selected approach for achieving a social goal. Interventions that use these strategies to build children's capacity to actively 
engage in social problem solving have the most significant and lasting impact on the quality of their social relationships (Greenberg, Kusche, 
& Riggs, 2004; Kam, Greenberg, & Kusche, 2004; Suveg, Southam-Gerow, Goodman, & Kendall, 2007). Examples for specific social skills 
addressed in each episode and examples of their in-game interactions are shown in Table 1. 

An interdisciplinary team of psychologists, computer programmers, and artists collaborated to create an engaging learning environment 
to serve as the context for social problem solving. We decided to develop Adventures Aboard the S.S. GRIN as a single player game rather than 
a multiplayer game for a number of reasons. A single player game enabled individualized feedback and play paths based on player choices, 
allowed for control over the reactions of other characters in a game, created a safe environment in which to practice fledgling skills without 
social ramifications, and avoided the possible iatrogenic effects of participants reinforcing negative behaviors in other children. 

The point-and-click problem solving tutorial provides a game-like feel due to the story arc, characters, and computer interaction. A 
compelling storyline and engaging challenges have been found to be critically important for engaging children in instructional software 
(Mott & Lester, 2006a, 2006b; Roussou, 2005), and this consideration was significant in the development of the program. The story world is 
based on a nautical adventure theme in which the player is a new recruit on the sailing ship S.S. GRIN and joins an established crew, each of 
whom specializes in a specific social skill (e.g., impulse control). The narrative purpose of the crew members is to be the child’s companions, 
but the tutoring system also employs them as pedagogical agents to provide feedback and assistance as needed. When the child first uses the 
software, he/she creates a customized avatar to represent himself or herself in the game. Using this avatar, the child then navigates the 
virtual environment, interacting directly with non-player characters (NPCs) and learning to appropriately manage social challenges. Spe- 
cifically, the child's avatar must join the crew and travel around an island engaging in social problem solving in order to address plot conflicts 
and save friendship on the island. 

As children navigate the program, the software captures in-game behaviors. Some of these behaviors are dialog-driven menu choices, 
whereas others are game-based behavior (e.g., if a child is asked to help with a task, does the child comply or choose another action, such as 
walking away?). The difficulty level of the social tasks increases both within and across the nine episodes of the game. For example, an early 
episode addresses relatively simple choices that show self-respect and respect for others, whereas a later episode addresses the skills 
needed to initiate positive social interactions, including the complex interaction of respect, observation, communication, thinking out 
options, and considering consequences. Within the “initiation” episode participants move from a relatively easy scenario (initiating with 
one non-intimidating other) to a more difficult one (initiating with two characters who are already engaged with each other) to an even 
more difficult scenario (initiating with a group of friends who appear intimidating and reject initial attempts). 


Table 1 

Description of episodes 1 -4 of the Adventures Aboard the S.S. GRIN ITS. 


Episode 

Topic 

Goals 

1 

Respect 

Demonstrate respect for self and 
others. 

2 

Consequences, 
Role Models, 
and Goals 

Identify long- and short-term 
consequences. Identify good role 
model qualities. Create and use an 
action plan to reach a goal. 

3 

Responsibility 
and Stop and 
Think 

Identify and follow through with 
responsible actions. Stop and think 
about how consequences relate to 
choices. 

4 

Communication 

Demonstrate understanding of 
verbal and nonverbal 
communication. 


Examples 

- Players teach one character how to respectfully interact with another. 

- Players face two challenges (one they successfully complete and one they do not) 
and learn to accept help and appropriately express pride. 

- Players are given a tempting but irresponsible choice. Later in the episode, they 
experience the consequence of their decision. 

- Players program a role model robot by choosing whether or not a variety of 
statements represent qualities of a good role model. 

- Players are given a responsibility and then choose to either follow through 
or deflect the responsibility onto another character. 

- Players help two characters resolve a conflict by matching several possible actions 
with their likely consequence, and then choosing the best option. 

- Players solve a puzzle that requires listening carefully to another player for clues. 

- Players navigate a maze by attending to body language, facial expression, and tone 
of voice to ascertain meaning. 
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As an ITS, it was essential that the software track players' actions, tailor hints and feedback along the way, collect information on per- 
formance, make inferences about players' strengths and weaknesses, adapt to the needs of the learners, and contain the following com- 
ponents: knowledge of the domain, knowledge of the learner, knowledge of appropriate pedagogy, and interactive environment interface 
(Samuelis, 2007). Central to the development of the program, we ensured that the program adapted to the needs of the players by 
monitoring progress in real time and adjusting gameplay accordingly. The software monitored individual progress and responded with 
appropriate content, feedback, and pedagogical assistance based on player actions. We incorporated proven training strategies of scaf- 
folding, coaching and reinforcement, and practice to create an effective social learning program for children. For example, in one task the 
player needs to complete a mission that initially appears possible to complete alone, but reveals itself to be a two-person job. Without 
explicit instruction, the player needs to seek the cooperation of an NPC. If the player does not perform this action in a given amount of time, 
an NPC walks through the scene as a hint that help is available. If the player still does not seek assistance after a given amount of time, the 
NPC reminds him or her that help is available. Finally, if the player is still attempting the impossible task, the NPC reappears to help and 
provides information about cooperation and help-seeking. At appropriate times, individualized feedback is provided in response to the 
child's performance, and when gameplay indicates that a skill has not been satisfactorily demonstrated the child is provided structured 
guidance, and in some cases is asked to continue to practice the skill in-game until he or she demonstrates an understanding of a good social 
choice. Achievements are provided for outstanding performance, and players are able to replay episodes to earn all achievements. All text- 
based dialog is accompanied by audio voiceover to account for differences in reading ability and to convey important social information, 
such as emphasis and tone of voice. 

In order to test the potential promise of this innovative computerized method for SST, we examined the impact of participation in the 
first half of the software through this study. In particular, we were interested in examining how children and families use the program 
outside of a clinical environment in order to determine its feasibility, usability, and potential efficacy as a broad-scale SST tool. The tested 
portion of the ITS included four instructional episodes, each of which focused on a specific social skill (i.e., respect, social planning, impulse 
control, and communication). Children completed 38 rooms (i.e., discrete visual spaces), interacted with 42 unique characters, and 
completed 14 social problem-solving scenarios for this study, totaling approximately 90 min of gameplay. Sample screenshots from Ad- 
ventures Aboard the S.S. GRIN are shown in Fig. 1 (further examples available at http://www.3ctechnologies.com/games). 

Our research questions were: 

1. Will children with social skills deficits find the ITS likeable and usable? 

2. Will interaction with the ITS be associated with reduced pychosocial distress and greater behavioral and emotional strength? 

We hypothesized that both research questions would be answered in the affirmative. 

2. Material and method 

2.7. Participants 

Participants were recruited in the spring of 2013 via postings on local parenting listservs, social media sites, and through contacts within 
area school systems and child service providers. Interested parents completed an online eligibility survey which included demographic 
information and the Behavior Assessment System for Children, Second Edition (BASC-2; Reynolds & Kamphaus, 2004), a behavior rating 
scale. To be eligible to participate, participants must have (1 ) been attending 3rd, 4th, or 5th grade in the fall of 2013; (2) received instruction 
in a regular education classroom for at least 40% of the school day; (3) been English language proficient; (4) had access to a computer with 
Internet use at home; and (5) scored at or above the clinical cut-off for their age and sex on any of 14 subscales on the BASC-2, or scored in 
the "at-risk” range on at least two subscales. 

To reduce sampling error, eligible participants were stratified by grade, sex, race, and scores on three BASC subscales (social skills, 
withdrawal, and aggression). The social skills subscale measures skills children need when interacting with peers and adults, including 
productive communication and offering to help others in need. The withdrawal subscale examines the tendency to isolate oneself from 
social interaction, including refusing to talk and preferring to be alone. Finally, the aggression subscale assesses the tendency to do or 
threaten physical or emotional harm to others (Reynolds & Kamphaus, 2004). These three subscales were included for stratification to 
balance the treatment and control groups because they represent adaptive/maladaptive and internalizing/extemalizing social behaviors. 
Following stratification, participants were randomly assigned to either an immediate treatment condition or a wait-list control condition. 
Study attrition, defined as failure to complete all four episodes and the pre- and post- test assessments, was 14% (n = 6) over the entire data 
collection period. Chi-square tests revealed no significant differences between participants who completed the program versus those who 
did not on grade, sex, race, household income, or BASC-2 subscale scores. 

Descriptive information regarding the participants for this study (n = 36) is presented in Table 2, separately for participants in the 
treatment and control groups. Chi-square tests revealed no significant differences between participants in the two study conditions for any 
variable other than race (x 2 = 3.955, p < .05). In the treatment group, there were more white participants than non-white (79% and 21%, 
respectively), whereas in the control group there were slightly more non- white participants than white (53% and 47%, respectively). 

2.2. Measures 

2.2.1. Usability and likability 

Following interaction with each episode of the program, children provided feedback on their experience via an online product evaluation 
survey. The surveys included pictures from the episode for reference, and children were asked to report how true for them were statements 
such as "Overall, I liked this whole episode of Adventures Aboard the S.S. GRIN," "I thought it was easy to understand how to make my 
character move or talk,” and "I think my friends would like this episode.” Response choices were presented on a five-point Likert-type scale 
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Fig. 1. Screenshots from Adventures Aboard the S.S. GRIN. 


ranging from 1 (NO! ) to 5 (YES! ), utilizing faces along with words to represent response options in a developmentally appropriate manner 
(see Fig. 2). For interpretation and analysis, the response choice scale was translated so that 1 = strongly disagree and 5 - strongly agree. 

2.2.2. Psychosocial distress 

The Youth Outcomes Questionnaire, Second Edition (Y-OQ2.01 ; Burlingame et al., 2005) is a 64 item parent-report measure designed to 
assess a child's psychosocial distress. The Y-OQ 2.01 consists of six subscales: Intrapersonal Distress, Somatic, Interpersonal Relations, 


Table 2 

Comparison of demographics and BASC-2 scores by intervention condition. 



Treatment n = 19 

Control n = 17 

Demographics 



Grade: % 3rd/4th/5th 

32/36/32 

41/24/35 

Sex: % F/M 

26/74 

28/72 

Race: % White/Non-White 

79/21 

47/53 

HH Income: % <$100K/S100K-$200K/>$200K 

59/29/12 

80/20/0 

BASC-2 Scores 



Social skills: % Average/At Risk/Clinical 

53/31/16 

52/24/24 

Withdrawal: % Average/At Risk/Clinical 

16/26/58 

29/24/47 

Aggression: % Average/At Risk/Clinical 

42/42/16 

41/41/18 


p < .05 for Treatment vs. Control condition. 
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NO! 


Fig. 2. Visual representation of response options. 


Critical Items, Social Problems, and Behavioral Dysfunction. Parents were asked to report how true statements (e.g., “Seems anxious or 
nervous,” “Doesn't have or keep friends,” “Acts impulsively, without thinking of the consequences”) were for their child during the past 
seven days on a 5-point Likert-type scale ranging from 0 (never or almost never) to 4 (almost always or always). Scores for each subscale are 
computed by summing the scores for all items in the subscale. The Y-OQ.2.01 Total is a composite score computed by summing scores on all 
subscales. Higher scores on the Y-OQ.2.01 Total and on each subscale indicate higher levels of psychosocial distress. The Y-OQ2.01 has high 
reliability (Cronbach's a = .97) and its publishers detail several studies establishing the content, criterion-related, and construct validity 
with both clinical and non-clinical (community) samples (Burlingame et al., 2005). We selected it to measure the effects of the ITS because of 
its utility as a global measure of observed behavior change over relatively short periods of time (Burlingame et al., 2005). 


2.2.3. Behavioral and emotional strength 

The Behavioral and Emotional Rating Scale, Second Edition (BERS-2; Epstein, 2004) is a 57 item strength-based assessment that mea- 
sures the positive emotions, behaviors, and aspects of an individual's life. The BERS-2 consists of five subscales: Interpersonal Strength, 
Family Involvement, Intrapersonal Strength, School Functioning, and Affective Strength. Parents were asked to think about their child's 
status over the past thirty days and report how much statements such as “participates in family activities,” “identifies own feelings,” and 
“shares with others" were like their child. Response choices were on a 4-point Likert-type scale ranging from 0 (not at all like your child) to 3 
(very much like your child). Raw scores for each subscale are computed by summing the scores for all items in each scale. Subscale raw 
scores are then converted into standard scores. Finally, the BERS-2 composite score, called the Strength Index, is calculated by summing the 
scaled scores and converting to a standard score. Higher scores on the subscales and on the composite Strength Index reflect greater 
behavioral and emotional strength. The BERS-2 has demonstrated very good reliability (Cronbach's a = .97), and factor analyses have 
established content-description, criterion-prediction, and construct-identification validity (Epstein, 2004). We selected the BERS-2 for this 
study because of its utility as a measure of treatment effects for interventions designed to enhance a child's behavioral and emotional 
strengths (Epstein, 2004). 


2.3. Procedure 


Precautions were taken to ensure study ethics and protection of human subjects. The study protocol was approved by an institutional 
review board, and verbal assent (from children) and written informed consent (from parents) was obtained from all participants prior to 
participation. All parts of the study were completed online through the secure project website. Prior to completing pretest measures and 
beginning the intervention, all participants viewed a welcome video explaining the study design and timeline, and training them in the use 
of the project website and the Adventures Aboard the S.S. GRIN computer program. 

All parents completed the BERS-2 and YOQ.2.01 as pretest measures. The week following pretest data collection, the treatment group 
began interacting with the Adventures Aboard the S.S. GRIN ITS. Episodes were released weekly for four weeks, and children had one week to 
complete each episode. There were three steps to each week’s activities: (1) participants played through the episode, (2) participants 
completed the episode quiz and mini-game, and (3) participants completed the product evaluation survey. The episode quiz reviewed the 
concepts and skills covered in each episode, and by playing the quiz participants were able to earn tokens which they used to play mini- 
games. Mini-games were included to increase engagement and provide a reward to children for completing the episode and quizzes. 

Following the treatment group's four week interaction with the ITS, parents in both groups completed the BERS-2 and the YOQ2.0 again. 
Once posttest data collection was completed, the wait-list control group received the Adventures Aboard the S.S. GRIN computer program to 
interact with as they wished. 

3. Results 

Our first research question was whether children with social skills deficits would find Adventures Aboard the S.S. GRIN likeable and 
useable. If children were unable or unwilling to independently and effectively use the program, its value as a social skills tutorial would be 
greatly reduced. We hypothesized that usability and likability would both be rated highly by participants, and these hypotheses were 
supported. Results are shown in Table 3 and described below. 

3.1. Usability 

Children found the program to be understandable and usable, supporting our first hypothesis. It was important to ensure that children 
understood how to interact with Adventures Aboard the S.S. GRIN because the program would be ineffective as a social skills intervention tool 
designed for nonclinical settings if the program design precluded its independent use by children. For the purposes of this study, children 
answered a series of questions related to the ITS mechanics and their understanding of the program. Children answered the following 
questions regarding usability for each part of each episode, and for each episode overall: (1)1 thought it was easy to understand what to do 
next: (2) I thought it was easy to understand what the characters were saying; (3) I thought it was easy to understand how to make my 
character move or talk. Results showed that children understood the program in terms of gameplay mechanics, the characters’ speech, and 
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Table 3 

Descriptive statistics for treatment group participants (n = 19). 
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M 

SD 

Usability 

Understood characters' speech 

4,45 

.23 

Understood how to move and talk 

4.47 

.21 

Understood what Avatar should do next 

4.18 

,13 

Ukabililty 

Liked program overall 

4.39 

.18 

Thought program was fun to play 

4.46 

.15 

Liked characters 

4.40 

.09 

Liked art and graphics 

4.47 

.07 

Would like to play more programs like this 

4.40 

.14 

Friends would like to play this program 

4.12 

.14 

Would like to play this program again 

4.05 

.11 


Note. The usability and likability values represent combined mean percentages for the first four episodes of the 
program. 1 = strongly disagree, 2 = disagree, 3 = Neither agree nor disagree; 4 = agree; 5 = strongly agree. 


how to select their Avatar's speech and actions. These findings assured us that the gameplay was not an obstacle to successful transmission 
of social skills instruction. 

3.2. Likeability 

Children also enjoyed the program, supporting our second hypothesis. The degree to which children enjoy an ITS such as Adventures 
Aboard the S.S. GRIN affects whether and how much they interact with the program. Because this program was designed to be played by 
children independently outside of a clinical setting, and because greater practice of social skills (i.e., dosage) is associated with greater skills 
gains, it was important for us to determine whether children enjoyed their interactions with the program. Immediately after completing 
each episode, children answered seven questions to assess the degree to which they liked interacting with the program: (1) I liked this 
episode: (2) I thought this episode was fun to play; (3) I liked the characters; (4) I liked the pictures/graphics: (5) 1 would like to play more 
programs like this one; (6) My friends would like to play this episode; (7) I would like to play this episode again. Results revealed that 
children enjoyed the program very much and gave the program high scores across all likability areas assessed. Knowing that the participants 
both understood and liked the program, we next turned our attention to the efficacy data. 

3.3. Psychosocial distress 

Once the usability and likeability aspects of the program were established, we investigated our second research question: whether 
interaction with the ITS would be associated with reduced psychosocial distress and increased behavioral and emotional strength. We 
hypothesized that they would be, and our hypothesis was supported. Descriptive statistics are presented in Tables 4 and 5, and results are 
described below. 

We examined the relation between interacting with the program and psychosocial distress scores through the following three methods. 
3.3.7. Significant change scores on the Y-OQ 2.01 

To highlight the practical significance of our findings, we calculated the number of children within each group who had significant 
changes in scores as provided in the measure's manual across each of the six subscales of the Y-OQ. 2.01, and for the total Y-OQ 2.01 


Table 4 

Percentage of participants in each group demonstrating significant changes on the Y-OQ 2.01. 


Y-OQ 2.01 subscale 

Treatment group (n = 19) 

Control group (n = 17) 

Intrapersonal distress 

Improved 

42% 

35% 

Deteriorated 

0% 

0% 

Somatization 

Improved 

11% 

6% 

Deteriorated 

0% 

0% 

Interpersonal relations 

Improved 

21% 

12% 

Deteriorated 

11% 

0% 

Social problems 

Improved 

0% 

0% 

Deteriorated 

0% 

0% 

Behavioral dysfunction 

Improved 

29% 

12% 

Deteriorated 

0% 

0% 

Critical items 

Improved 

26% 

24% 

Deteriorated 

0% 

6% 

Y-OQ2.01 Total score 

Improved 

68% 

52% 

Deteriorated 

0% 

0% 
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Table 5 

Percentage of participants in each group demonstrating significant changes on the BERS-2. 


BERS-2 subscale 

Treatment group (n = 19) 

Control group (n = 17) 

Interpersonal strength 

Improved 

26% 

24% 

Deteriorated 

0% 

12% 

Family involvement 

Improved 

16% 

12% 

Deteriorated 

0% 

12% 

Intrapersonal strength 

Improved 

26% 

0% 

Deteriorated 

0% 

0% 

School functioning 

Improved 

26% 

12% 

Deteriorated 

26% 

18% 

Affective strength 

Improved 

42% 

.12% 

Deteriorated 

0% 

6% 

Total strength index 

Improved 

16% 

12% 

Deteriorated 

0% 

18% 


psychosocial distress index. “Significant change” is a reliable change index used to determine if the change exhibited by an individual is 
clinically significant; individuals' scores must change by a minimum amount based on analyses of norm data by the measure's developer in 
order to be considered significantly different from pretest to posttest. These results are displayed in Table 4. Relative to the control group, 
more children in the treatment group had significant improvement across each subscale and on the overall Y-OQ 2.01 index score. 

3.3.2. Testing treatment effects on the Y-OQ 2.07 with ANCOVA 

Outcomes for the treatment and control group participants were compared using a one-way ANCOVA, with treatment condition as the 
independent variable (i.e., the treatment group completed four episodes of Adventures Aboard the S.S. GRIN over a four-week period, and the 
control group did not interact with the program) (Cohen, 2001; Field, 2005; Keselman et al., 1998). There is substantial support in the 
literature for using ANCOVA when groups do not differ significantly on the covariate (e.g., Miller & Chapman, 2001 ) and when the research 
question is whether posttest means, adjusted for pretest scores, differ between the two conditions (Sweet & Grace-Martin, 2011). Posttest 
Total Score on the Y-OQ 2. 01 was the dependent variable, with pretest Total Score on the Y-OQ 2.01 as the covariate. All tests were conducted 
using Sidak adjusted alpha levels to account for multiple comparisons (Field, 2005). 

The co variate, Y-OQ 2.01 Total Score at pretest, was significantly related to parent-reported psychosocial distress at post-test, F(l, 
34) = 194.285, p < .001, r = .923, showing the expected finding that children with higher scores at pretest also had higher scores at posttest. 
We included pretest score as the covariate to remove the variance associated with pretest scores and determine whether playing Adventures 
Aboard the S.S. GRIN contributed significantly to the variance in Y-OQ 2.01 Total Scores at posttest when controlling for pretest scores. We 
found that participants in the treatment group evidenced significantly lower psychosocial distress at post-test (M = 23.32, SD = 18.79), 
compared to those in the control group (M = 48.71, SD = 42.16) after controlling for pretest psychosocial distress scores, F{ 1, 34) = 4.131, 
p = .05. 

Lastly, we calculated effect sizes using t- statistics, based on Field's (2005) recommendation against using n 2 or oj 2 with unequal numbers 
of participants in the two experimental conditions. The strength of the relationship between playing Adventures Aboard the S.S. GRIN and Y- 
OQ2.01 posttest scores was moderate (r = .329), according to Cohen's (1992) conventions. This result, combined with the change scores 
presented in Table 4, indicates that interaction with the program led to lower psychosocial distress. 

3.4. Behavioral and emotional strength 

We followed the same statistical procedures as for the Y-OQ 2.01 analyses, first comparing the numbers of participants in each treatment 
condition who exhibited a significant pretest to post- test change in scores on the BERS-2, and then conducting a one-way ANCOVA to further 
investigate the effects of playing Adventures Aboard the S.S. GRIN. Alpha levels were again Sidak adjusted to account for multiple compar- 
isons, per Field's (2005) recommendation. 

3.4.1. Significant change scores on the BERS-2 

We calculated the number of children within each group who had statistically significant changes in scores across each of the five 
subscales of the BERS-2, and for the BERS-2 Total Strength Index. Per the BERS-2 manual, this indicates the children within each group who 
had significant changes in scores as determined by the standardized test scores and the normal distribution (i.e., scores must change by 
more than one standard deviation from the mean to be determined significantly different from pretest to posttest). As shown in Table 5, 
more children in the treatment group than in the control group had significant improvements across all subscales and on the overall BERS-2 
index score. With the exception of the School Functioning subscale, no children in the treatment group exhibited significant deterioration in 
any areas assessed by the BERS-2, whereas in the control group there were children with significant declines in scores across every subscale. 
It should be noted that because all data collection occurred in the summer, school functioning results should be interpreted with caution. 

3.4.2. Testing treatment effects on the BERS-2 with ANCOVA 

A one-way ANCOVA was conducted to assess the relation between playing Adventures Aboard the S.S. GRIN and behavioral and emotional 
strength. The independent variable, treatment condition, included the treatment level and control level. The dependent variable was 
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parent-reported emotional and behavioral strength at posttest, as measured by the BERS-2 Total Strength Index. The co variate, BERS-2 Total 
Strength Index pretest score, was significantly related to parent-reported emotional and behavioral strength at post-test, F(l, 34) = 74.422, 
p < .01, r = .686, indicating that, as expected, children's scores at pretest were related to their scores at post-test. We included pretest scores 
on the BERS-2 Total Strength Index as a covariate in order to account for this confounding variable and obtain a more pure measure of the 
effect of this ITS. 

Children who played Adventures Aboard the S.S. GRIN showed higher levels of emotional and behavioral strength at post-test (M = 93.21, 
SD = 13.41) relative to children who did not interact with the program (M = 86.94, SD = 19.61 ) after controlling for the pre-test strength 
scores, F[ 1, 34) = 5.352, p < .05, r = .136. This finding, combined with the change scores presented in Table 5, indicates that interaction with 
Adventures Aboard the S.S. GRIN led to significant improvement in emotional and behavioral strength. This effect size represents a small effect 
according to Cohen's (1992) conventions. 

4. Discussion 

The goal of the current study was to investigate the viability of leveraging gaming technology to advance mental health service delivery 
by reaching beyond current in-person SST delivery models. We tested whether children with social skills challenges in grades 3—5 would 
like and be able to independently use the program Adventures Aboard the S.S. GRIN, and whether use of the program was associated with 
positive changes in psychosocial distress and behavioral and emotional strength. The instructional goals and approach of the program were 
based on SSGRIN, an evidence-based, in-person group social skills program designed to help children who struggle with peer relationships 
learn the skills needed to enhance their social behavior and relationships. The in-person program has been proven successful in small group 
settings when administered by a trained professional (DeRosier, 2004; DeRosier & Gilliom, 2007; DeRosier & Marcus, 2005), and our goal 
was to extend the effective delivery of this program through the use of technology. By using a randomized controlled trial design, we were 
able to gather evidence of the promise of this new intervention modality for enhancing social learning and behavior. 

A persistent challenge in the design and creation of educational games is the balance between entertainment, demandingness, and 
instruction. The acceptability of a program, as measured by likeability and usability, is important to assess because if the program isn’t 
enjoyable and engaging, children won’t meaningfully interact with it; if the level of challenge is too low or too high, children will either be 
bored or frustrated and learning will be diminished. Program efficacy, as measured by knowledge and/or behavior change is also important 
to evaluate, because if the program is too entertaining there is a risk of the educational goals being overshadowed by the entertainment 
aspects of the game. Due to the innovative nature of using ITSs for SST, it was important that we assess both acceptability and efficacy of 
Adventures Aboard the S.S. GRIN. 

We found that children rated the ITS very highly on acceptability for both likeability and ease of use. These results were highly 
encouraging, because the nonclinical setting delivery of this program requires child program users to be motivated and able to use the 
software with little or no intervention in order for it to be effective. The mean ratings for each of the items assessing the extent to which 
children found the ITS enjoyable and easy to use were between “agree" and "strongly agree." These findings indicate that the program was 
engaging and appealing to children in grades 3-5. Children indicated that they would like to play the game again, which is encouraging 
because practicing appropriate social skills is a factor in successful generalization of the skills to new settings. 

We also found support for the efficacy of the program, in that interaction with the program was associated with positive, observable 
behavior change. Children who played Adventures Aboard the S.S. GRIN evidenced decreased psychosocial distress and increased behavioral 
and emotional strength relative to children who did not interact with the program. The effect size for psychosocial distress was moderate, 
and the effect size for behavioral and emotional strength was small but significant. This significant effect is especially notable given the novel 
intervention modality of the program and the low dose of intervention. In addition, examination of significant change scores for each 
subscale of these constructs showed that more children in the treatment group showed positive change on the subscales of intrapersonal 
distress, somatization, interpersonal relationships, social problems, behavioral dysfunction, critical items, family involvement, school 
functioning, and affective strength. Taken together, these results provide evidence that children with social skills challenges can benefit from 
playing Adventures Aboard the S.S. GRIN. 

4.1. Intelligent tutoring systems for social skills training 

The results of this study add to an established body of work showing that effective SST can foster positive behavioral and emotional 
outcomes. Our findings extend prior research by demonstrating that a game-like ITS can be an effective vehicle for conveying this training. 
Evidence for the effectiveness of ITSs for SST is growing, although programs tend to focus on more basic skills such as memorization of faces 
and emotions as opposed to more complex social problem solving scenes, or are limited by experimental design issues. 

Our results corroborate and extend the existing evidence in this emerging, innovative field. Preliminary evidence on two bullying 
prevention interactive computer games hint at the promise of this medium for effective social skills training. Evidence from Quest for the 
Golden Rule (Rubin-Vaughan et al., 2011) shows that children who interacted with the game gave high ratings for enjoyment, and had 
greater knowledge of bullying and improved identification of strategies to prevent bullying at post-test than at pretest. The quasi- 
experimental design of the Rubin-Vaughan et al. study, however, prevents causal conclusions about the role of the game in these 
changes. Evaluation of the effectiveness of the virtual role playing bullying prevention program, FearNot! (Hall, Woods, Hall, & Wolke, 2007), 
did employ a true experimental design and found a decrease in bullying victimization for children who interacted with the program. Our 
findings add to the body of work in the emerging field of ITS for SST by using a randomized controlled trial design to provide evidence that a 
SST designed to deliver an evidence-based program for general social skills improvement can lead to observable behavior change, that the 
program is well-liked by users, and that it is easy for children to use independently outside the clinical setting. 

ITSs offer a number of advantages over traditional SST methodologies. As mentioned previously, ITS s mitigate several of the challenges 
associated with traditional in-person SST (e.g., time, cost, and accessibility) that limit the feasibility of in-person treatment for many in- 
dividuals. In addition to these practical considerations, ITSs offer other advantages. Children are more engaged and motivated to learn when 
interacting with educational games as compared to traditional learning environments (Annetta, Minogue, Holmes, & Cheng, 2009; Chandra 
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& Lloyd, 2008), and a moderate amount of interaction with pro-social computer game content is beneficial to children's academic, cognitive, 
and social skills (Tran & Subrahmanyam, 2013). Huffert and Calvert (2003) note that because ITSs motivate children to engage in the learning 
process and encourage self-direction and the application of learning to new situations, they support the critical components of learning. 
Because ITSs can be presented in an interactive format that presents multiple examples of content over varying levels of difficulty and 
provides immediate feedback and guidance based on participant responses (Hutcherson et al., 2004; Mazzotti et al., 2009), children are able 
to work independently and can repeatedly practice scenarios. ITSs also provide a safe environment in which to try out and practice new or 
different social skills, a task that may be uncomfortable for children to engage in with peers. Working on social skills with peers, even in 
group therapy sessions, leaves children open to the potentially negative reactions of other children. In ITSs, the reactions of the characters in 
the program can be controlled so that children are not discouraged from engaging and practicing skills. Emerging technologies may be a 
viable answer to the challenges of maintenance and generalization associated with social skills interventions, and may motivate learning 
(Fitzgerald, 2005; Goldsworthy, Barab, & Goldsworthy, 2000). 

4.2. Limitations & future directions 

This pilot study offers important insights about the feasibility of the online social skills training Adventures Aboard the S.S. GRIN ITS, and 
the efficacy results are very encouraging for the promise of technology-based social skills intervention. It should be noted that efficacy tests 
are not intended to prove full effectiveness of an intervention, and future development and testing of this promising program will provide 
more information about its capabilities. Because this study was designed as a pilot efficacy test, our sample size was relatively low, resulting 
in limited statistical power. As a result, we were constrained to fairly basic statistical methods and were unable to examine potentially 
informative interactions with variables such as age, sex, and specific social skill deficit. Another consideration is that without an alternate 
activity for the waitlist control group, the placebo effect is a possibility for observed behavior changes in children in the treatment group. In 
addition, because the pilot test was conducted over the summer we were unable to assess in-school behaviors. Finally, this study was 
conducted on the first half of the complete tutorial and as a result we were unable to examine the effects of the full program. 

The next step in our research will be the development and large-scale testing of the complete ITS. An additional five instructional ep- 
isodes targeting specific social skills (i.e., checking assumptions, initiating friendships, cooperation, compromise, and dealing with peer 
pressure and emotional interference) will complete the program. Once the full ITS is developed, we will conduct a large-scale efficacy trial 
that will include teacher ratings of children's behavior. Combined with parent and child data, this information will give us a comprehensive 
understanding of the effects of the intervention. This larger efficacy test will also include a substitute online activity for participants in the 
waitlist group (e.g., interaction with an academic website) in order to reduce the possibility of placebo effects among the group receiving the 
intervention. 

In the follow-up study with greater statistical power we will also examine effects of the intervention for children of different ages, and for 
different subtypes of socially struggling children (e.g., aggressive, withdrawn, rejected). In addition, we will explore sex differences in the 
use, appeal, and effectiveness of the program. We will also examine dosage effects to determine whether additional exposure has added 
benefits, and if so, for whom. In future studies we also plan to include additional behavioral, emotional, and academic outcome variables to 
investigate the effects of the intervention across a broad range of child outcomes. A final step in evaluating the efficacy of Adventures Aboard 
the S.S. GRIN will be a comparative study. We will examine how the online program compares not only to a control group, but also to the in- 
person SSGRIN, and to a combined SST that includes both the in-person and online aspects of the program. This will enable us to determine 
the added value of the ITS. 

A future phase of our research will include investigating the long term effects of the intervention. The developmental nature of social 
relationships means that some positive effects of SST may be evidenced after a time delay. For example, an impulsive child who learns to 
stop and think before reacting can, over time, earn the trust and respect of others. Peers, teachers, and parents can reinforce this appropriate 
behavior. This may lead to greater peer acceptance, improved self-esteem and family relationships, and better social and academic func- 
tioning over time. An investigation of the long term effects of the in-person SSGRJN program revealed that immediate gains were main- 
tained, and additional positive treatment effects emerged one-year following program participation (DeRosier & Marcus, 2005). Specifically, 
children who participated in SSGR/N evidenced higher social acceptance and self-esteem, and lower depression and anxiety relative to the 
study's control group. Because Adventures Aboard the S.S. GRIN enables unlimited exposure and practice opportunity, we expect that the 
longer term effects of this ITS may be even greater than the in-person program. 

5. Conclusion 

Developing evidence-based ITSs for SSTs involves the integration of developmental theory, research, computer science, gaming theory, 
and practical application to improve social programs and interventions for children. The results of this study provide evidence that ITSs can 
be an effective and welcomed strategy for addressing psychosocial and behavioral issues in the social domain. This methodology has the 
potential to overcome many of the barriers associated with traditionally-delivered SSTs, and as a result can deliver services much more 
broadly than existing methodologies. ITSs can be made available to any individual or organization (e.g., educators, parents, mental health 
providers, youth groups) and can be used in any setting with a computer (e.g., schools, homes, clinics, after-school programs). 

Translating evidence-based social skills intervention into ITSs creates the opportunity to provide widespread access to social inter- 
vention, and thus close the gap between proven interventions and their widespread use by those most in need. Expanding access to proven 
programs in this way has important implications for fostering optimal development and promoting positive development for children at 
risk for poor social, emotional, and academic outcomes. 

Acknowledgments 

This research was supported by a grant from the Institute for Education Science (Grant #R305A110583) to the Center for Research in 
Emotional and Social Health (CRESH). We gratefully acknowledge their financial support. We also thank our development team for their 


R.P. Sanchez et al / Computers & Education 78 (2014) 321-332 


331 


contributions to the program, particularly Chris Hehman, Jeremiah Weatherley, Matt Habel, Chrystin Garland, Jon Fabris, Charles Bevan, 
Marcelo Nader, Kim Smith, and Wes Sommer. 


References 


Annetta, L. A., Minogue, J., Holmes, S. Y., & Cheng, M. (2009). Investigating the impact of video games on high school students' engagement and learning about genetics. 
Computers & Education, 53, 74-85. 

Asher, S. R., Parker, J. C., & Walker, D. L (1996). Distinguishing friendship from acceptance: implications for intervention and assessment. In W. M. Bukowski, A. F. Newcomb, & 
W. W. Hartup (Eds,), The company they keep: Friendship in childhood and adolescence (pp. 366-405). New York, NY: Cambridge University Press. 

Asher, S. R., & Renshaw, P. D. (1981). Children without friends: social knowledge and social-skill training. The Development of Children's Friendships, 2, 273. 

Bandura, A. (1977). Self-efficacy: toward a unifying theory of behavioral change. Psychological Review, 84(2), 191-215. 

Beidel, D., Turner, S„ Young, B., & Paulson, A. (2005). Social effectiveness therapy for children: three-year follow-up. Journal of Consulting and Clinical Psychology, 73, 721-725. 

Bloomquist, M. L. (2005). Skills training for children with behavior disorders , revised edition: A parent and therapist guidebook. New York, NY: The Guilford Press. 

Bordeau, J., & Grandbastien, M. (2010). Modeling tutoring knowledge. In R. Nkambou, J. Bourdeau, & R. Mizoguchi (Eds.), Advances in inte//igent tutoring systems (pp. 123-143). 
Berlin: Springer-Verlag. 

Burlingame, G. M., Wells, M. G„ Cox, J. C., Lambert, M. J„ Latkowski, M., & Justice, D. (2005). Youth Outcome Questionnaire. Salt Lake City, UT: OQ, Measures, LLC, 

Bushaw, W. J., & Lopez, S. J. (2013). Which way do we go? The 45th annual PDK/Callup Poll of the public's attitudes toward the public schools, http://pdkintl.org/noindex/2013_ 
PDKGallup.pdf Accessed 11.22,13. 

Camilli, G., Vargas, S., Ryan, S., & Barnett, W. S. (2010). Meta-analysis of the effects of early education interventions on cognitive and social development. Teachers Cotlege 
Record, 112, 579-620. 

Chandra, V., & Lloyd, M. (2008). The methodological nettle: 1CT and student achievement. British Journal of Educational Technology, 39, 1087—1098. 

Cohen, J. (1992). A power primer. Psycho/ogica/ Bulletin, 772(1 ), 155-159. 

Cohen, B. H. (2001). Explaining psychological statistics (2nd ed.). New York, NY: John Wiley & Sons, Inc. 

Coie, J. D., Dodge, K. A., & Kupersmidt, J. B. (1990). 2 Peer group behavior and social status. Peer Rejection in Childhood, 17. 

Collins, J. W„ & O'Brien, N. P. (2011). The Greenwood Dictionary of Education (2nd ed.). Westport, CT: ABC-CUO. 

Corbett, A., Koedinger, K., & Hadley, W. (2001). Cognitive tutors: from the research classroom to all classrooms. In P. S. Goodman (Ed.), Technology enhanced learning: Op- 
portunities for change (pp. 235-263). Mahwah, NJ: Erlbaum. 

Crick, N., & Dodge, K. (1994). A review and reformulation of social in formation- pro cessing mechanisms in children’s social adjustment. Psychological Bulletin. 175(1), 74-101. 

DeRosier, M. E. (2004). Building relationships and combating bullying: effectiveness of a school-based social skills group intervention. Journal of Clinical Child and Adolescent 
Psychology, 33, 196-201. 

DeRosier, M. E. (2007). Peer-rejected and bullied children: a Safe Schools Initiative for elementary school students (Chap. 15). In J. E. Zins, M. J. Elias, & C. A. Maher (Eds,), 
Bullying, victimization, and peer harassment: A handbook of prevention and intervention. Binghamton, NY: Haworth Press. 

DeRosier, M. E., & Gilliom, M. (2007). Effectiveness of a parent training program for improving children's social behavior. Jo urn al of Child and Family Studies, 16, 660-670. 

DeRosier, M. E., Kupersmidt, J. B., & Patterson, C. P. (1994). Children's academic and behavioral adjustment as a function of the chronicity and proximity of peer rejection. Child 
Development, 65, 1799-1813, 

DeRosier, M. E., & Marcus, S. R. (2005). Building friendships and combating bullying: effectiveness of S. S. GRIN at one-year follow-up. Journal of Clinical Child and Adolescent 
Psychology, 34, 140-150. 

Dodge, K. A., & Crick, N. R. (1990). Social information-processing bases of aggressive behavior in children. Personality and Social Psychology Bulletin, 16, 8-22. 

Dodge, K. A., & Feldman, E. (1990). Issues in social cognition and sociometric status. Peer Rejection in Childhood, 119—155. 

Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011 ). The impact of enhancing students' social and emotional learning: a meta-analysis of 
school-based universal interventions. Child Development, 82, 405-432. 

Dusenbury, L, Zadrizil, J„ Mart, A., & Weissberg, R. P. (2011 ). State learning standards to advance social and emotional learning. Chicago, IL: CASEL - Collaborative for Academic, 
Social, and Emotional Learning. 

Elliott, S., & Gresham, F. (1993). Social skills interventions for children. Behavior Modification, 17, 287—313. 

Epstein, M. H. (2004). Behavioral and Emotional Rating Scale (2nd ed.). Austin, TX: PRO-ED. 

Field, A. (2005). Discovering statistics using SPSS (2nd ed.). Thousand Oaks, CA: SAGE Publications Ltd. 

Fitzgerald, G. E. (2005). Using technologies to meet the unique needs of students with emotional/behavioral disorders: findings and directions. In D. Edyburn, K. Higgins, & 
R. Boone (Eds.), Handbook of special education technology research and practice (pp. 335-354). Whitefish Bay, Wl: Knowledge by Design, Inc. 

Foster, S. L., & Bussman, J. R. (2007). Evidence-based approaches to social skills training with children and adolescents. In R. G, Steele, T. D. Elkin, & M, C. Roberts (Eds.), 
Handbook of evidence-based therapies for children and adolescents: Bridging science and practice (pp. 409-428). New York, NY: Springer Science & Business Media, LLC. 

French, D. C., & Conrad, J. (2001). School dropout as predicted by peer rejection and antisocial behavior, journal of Research on Adolescence, 11, 225—244. 

Garris, R., Ahlers, R., & Driskell, J. (2002). Games, motivation, and learning: a research and practice model. Simulation & Gaming, 33(4), 441—467. 

Gee, J. P. (2003). What video games have to teach us about learning and literacy? ACM Computers in Entertainment, 1, 1—4. 

Golan, O,, & Baron-Cohen, S. (2006). Systemizing empathy: teaching adults with Asperger syndrome or high-functioning autism to recognize complex emotions using 
interactive multimedia. Development and Psychopathology, 18, 591-617. 

Goldsworthy, R., Barab, S. A., & Goldsworthy, E. L. (2000). The STAR Project: enhancing adolescents' social understanding through video-based, multimedia scenarios. Journal 
of Special Education Technology, 15, 13-26. 

Graczyk, P., Weissberg, R., Payton, J., Elias, M., Greenberg, M., & Zins, J. (2000). Criteria for evaluating the quality of school-based social and emotional learning programs. In 
R. Bar-On, & J. Parker (Eds.), The handbook of emotional intelligence; Theory, development, assessment, and application at home, school, and in the workplace (pp. 391—410). 
San Francisco, CA: Jossey-Bass. 

Graesser, A., Jeon, M„ & Duffy, D. (2008). Agent technologies designed to facilitate interactive knowledge construction. Discourse Processes, 45(4— 5), 298—322. 

Greenberg, M. T„ Domitrovich, C., & Bumbarger, B. (2001). The prevention of mental disorders in school-aged children: current state of the field. Prevention & Treatment 4 
1-67. 

Greenberg, M., Kusche, C„ & Riggs, N. (2004). The PATHS curriculum: theory and research on neuro cognitive development and school success. In J. Zins, R r Weissberg, 
M. Wang, & H. Walberg (Eds.), Building academic success on social and emotional learning; What does the research say? (pp. 170-188). New York, NY: Teachers College Press. 

HR 1875. (2013). Academic, Social, and Emotional Learning Act o/2073. Retrieved December 3, 2013, from https://www.govtrack.us/congress/bills/113/hrl875. 

Hall, 1., Woods, S., Hall, M., & Wolke, D. (2007). Children's emotional interpretation of synthetic character interactions. Affective Computing and Intelligent Interaction 4738 
642-653. 

Hawker, D. S., & Boulton, M. J. (2000). Twenty years' research on peer victimization and psychosocial maladjustment: a meta-analytic review of cross-sectional studies. 
Journal of Child Psychology & Psychiatry & Allied Disriplines, 47, 441—455. 

Hawkins, J., Catalano, R., & Miller, J. (1992). Risk and protective factors for alcohol and other drug problems in adolescence and early adulthood: implications for substance 
abuse prevention. Psychological Bulletin, 772, 64—105. 

Huffert, D. A., & Calvert, S. L (2003). The new science of learning: active learning, metacognition, and transfer of knowledge in e-leaming applications. Journal of Educational 
Computing Research, 29, 325—334. 

Hutcherson, K., Langone, J., Ayers, K., & Clees, T. (2004). Computer assisted instruction to teach item selection in grocery stores: an assessment of acquisition and gener- 
alization, journal of Special Education Technology, 19, 33—42. 

Joseph, G. E., & Strain, P. S, (2003). Comprehensive evidence-based social-emotional curricula for young children: an analysis of efficacious adoption potential. Topics in Early 
Childhood Special Education , 23, 65—76. 

Joyce, A., Gerhard, P,, & Debry, M. (Eds,). (2009). How are digital games used in schools: Complete results of the study. 

Kam, C., Greenberg, M., & Kusche, C. (2004). Sustained effects of the PATHS curriculum on the social and psychological adjustment of children in special education. Journal of 
Emotional and Behavioral Disorders, 12, 66—78. 

Ke, F. (2009). A qualitative meta-analysis of computer games as learning tools, in R. E. Furdig (Ed.), Handbook of research on effective electronic gaming in education (pp. 1-32). 
New York, NY: IGI Global. 


332 


R.P. Sanchez et aL / Computers & Education 78 (2014) 321-332 


Keselman, H. J., Huberty, C. J., Ux, L. M., Olejnik, S., Cribbie, R. A., Donahue, B., et al. (1998). Statistical practices of educational researchers: an analysis of their ANOVA, 
MANOVA, and ANCOVA analyses. Review of Educational Research, 68, 350-386. 

Kupersmidt, J. B., & DeRosier, M. E. (2004). How peer problems lead to negative outcomes: an integrative mediational model. In J. B. Kupersmidt, & K. A. Dodge (Eds.), 
Children's peer relations: From development to intervention (pp. 119—138). Washington, DC: American Psychological Association. 

Losel, F., & Beelmann, A. (2003). Effects of child skills training in preventing antisocial behavior: a systematic review of randomized evaluations. Annals of the American 
Academy of Political and Social Science, 587, 84-109. 

Mathan, S. A., & Koedinger, K. R. (2002). An empirical assessment of comprehension fostering features in an intelligent tutoring system. Berlin, Germany: Springer. 

Mazzotti, V. L, Test, D. W., Wood, C. L, & Richter, S. (2009). Effects of computer assisted instruction on student’s knowledge of post-school options. Career Development for 
Exceptional Individuals Retrieved from http://cde.sagepub.com/cgi/content/abstact/0885728809338714vl. 

Meier, C. R., DiPerna, J. C., & Oster, M. M. (2006). Importance of social skills in the elementary grades. Education and Treatment of Children, 29, 406—419. 

Merrell, W. K., & Gimpel, G. A. (1997). Soda/ skills of children and adolescents: Conceptualization, assessment and treatment. New York, NY: Guilford Press. 

Miller, G. A., & Chapman, J. P. (2001). Misunderstanding analysis of covariance. Journal of Abnormal Psychology, 110( 1), 40—48. 

Mott, B„ & Lester, J. (2006a). Probabilistic goal recognition in interactive narrative environments. In Proceedings of the Twenty-First National Conference on Artificial Intelligence 
(pp. 187—192). Boston: Massachusetts. 

Mott, B., & Lester, J. (2006b). Narrative-centered tutorial planning ro inquiry-based learning environments. In M. Ikeda, T. Chan, & K. Ashlay (Eds.), Eighth International 
Conference on Intelligent Tutoring Systems (pp. 675—684). Jhongli, Taiwan. 

Mueser, K. T., & Bellack, A. S. (2007). Social skills training: alive and well? Journal of Mental Health, 16, 549-552. 

Najaka, S. S., Gottfredson, D. C., & Wilson, D. B. (2001). A meta-analytic inquiry into the relationship between selected risk factors and problem behavior. Prevention Science, 2, 
257-271. 

National Association for the Education of Young Children. (2009). Developmentatly appropriate practice in early childhood programs serving children from birth through age 8. 
Washington, DC: NAEYC. 

National Research Council. (2012). Education for life and work: developing transferable knowledge and skills in the 21st century. In J. W (Ed.), Committee on Defining Deeper 
Learning and 21st Century Skills. Washington, DC: National Academies Press. 

NCREL & Metiri Group. (2003). enGauge 21st century skills: Literacy in the digital age. Naperville, IL: NCREL & Metiri Group. 

Olweus, D. (1993). Victimization by peers: antecedents and long-term consequences. In K. Rubin, & J. Asendorpf (Eds.), Social withdrawal, inhibition, and shyness in childhood 
(pp. 315-341). Mahwah, Nj: Erlbaum. 

Parker, J. G., Rubin, K. H., Erath, S. A., Wojslawowicz, J. C., & Buskirk, A. A. (2006). Peer relationships, child development, and adjustment: a developmental psychopathology 
perspective. In D. Cicchetti, & D. J. Cohen (Eds.), Developmental psychopathology (pp. 419—493). New York, NY: Wiley. 

Reynolds, C. R„ & Kamphaus, R. W. (2004). Behavior assessment system for children (2nd ed.). Circle Pines, MN: AGS Publishing. 

Roussou, M. (2005). Learning by doing and learning through play: an exploration of interactivity in virtual environments for children. ACM Computers in Entertainment, 2(1) 
1-23. 

Rubin-Vaughan, A., Pepler, D., Brown, S., & Craig, W. (2011). Quest for the golden rule: an effective social skills promotion and bullying prevention program. Computers & 
Education, 56, 166-175. 

Samuel is, L (2007). Notes on the components for intelligent tutoring systems. Acta Polytencia Hungaria, 4(2), 77—85. 

Sansoti, F. J., & Powell-Smith, K. A. (2008). Using computer- presen ted social stories and video models to increase the social communication skills of children with high- 
functioning autism spectrum disorders. Journal of Positive Behavior Interventions, 10, 162-178. 

Self, J. (1999). The defining characteristics of intelligent tutoring systems research: ITSs care, precisely, international Journal of Artificial Intelligence in Education, 10, 350—364. 

Solberg, M., Olweus, D., & Endresen, I. (2007). Bullies and victims at school: are they the same pupils? British youmal of Educational Psychology, 77, 441—464. 

Spooner, C. (1999). Causes and correlates of adolescent drug abuse and implications for treatment. Drug and Alcohol Review, 18, 453—475. 

Sweet, S. A., & Grace-Martin, K. A. (2011). Data analysis with SPSS: A _hrst course in applied statistics (4th ed.). Boston, MA: Pearson Education Inc. 

Suveg, C„ Southam-Gerow, M., Goodman, K., & Kendall, P. (2007). The role of emotion theory and research in child therapy development. Clinical Psychology: Science and 
Practice, 14, 358-371. 

Tran, P., & Subrahmanyam, K. (2013). Evidence-based guidelines for the informal use of computers by children to promote the development of academic, cognitive, and social 
skills. Ergonomics, 56, 1349-1362. 

Weiss, G,, & Hechtman, L. T. (1993). Hyperactive children grown up: ADHA in children, adolescents, and adults (2nd ed.). New York, NY: Guilford Press. 

Zins, J. E., Weissberg, R. P., Wang, M. C., & Walberg, H. J. (2004). Building school success through social and emotional learning. New York. 


